TaskMAD: a Platform for Multimodal Task-Centric Knowledge-Grounded Conversational Experimentation

The role of conversational assistants continues to evolve, beyond simple voice commands to ones that support rich and complex tasks in the home, car, and even virtual reality. Going beyond simple voice command and control requires agents and datasets blending structured dialogue, information seeking, grounded reasoning, and contextual question-answering in a multimodal environment with rich image and video content. In this demo, we introduce Task-oriented Multimodal Agent Dialogue (TaskMAD), a new platform that supports the creation of interactive multimodal and task-centric datasets in a Wizard-of-Oz experimental setup. TaskMAD includes support for text and voice, federated retrieval from text and knowledge bases, and structured logging of interactions for offline labeling. Its architecture supports a spectrum of tasks that span open-domain exploratory search to traditional frame-based dialogue tasks. It's open-source and offers rich capability as a platform used to collect data for the Amazon Alexa Prize Taskbot challenge, TREC Conversational Assistance track, undergraduate student research, and others. TaskMAD is distributed under the MIT license

The BladeMistress Corpus: From Talk to Action in Virtual Worlds

htmlabstractVirtual Worlds (VW) are online environments where people come together to interact and perform various tasks. The chat transcripts of interactions in VWs pose unique opportunities and challenges for language analysis: Firstly, the language of the transcripts is very brief, informal, and task-oriented. Secondly, in addition to chat, a VW system records users’ in-world activities. Such a record could allow us to analyze how the language of interactions is linked to the users actions. For example, we can make the language analysis of the users dialogues more effective by taking into account the context of the corresponding action or we can predict or detect users actions by analyzing the content of conversations. Thirdly, a joined analysis of both the language and the actions would empower us to build effective modes of the users and their behavior. In this paper we present a corpus constructed from logs from an online multiplayer game BladeMistress. We describe the original logs, annotations that we created on the data, and summarize some of the experiments

flexdiam – flexible dialogue management for incremental interaction with virtual agents (demo paper)

Yaghoubzadeh R, Kopp S. flexdiam – flexible dialogue management for incremental interaction with virtual agents (demo paper). In: Traum D, Swartout W, Khooshabeh P, Kopp S, Scherer S, Leuski A, eds. Proceedings of the 16th International Conference on Intelligent Virtual Agents (IVA 2016). LNCS (LNAI). Vol 10011. Berlin, Heidelberg: Springer; 2016: 476-479

Desparately Seeking Cebuano

At 4:13 A.M. Eastern Standard Time on Wednesday March 5, Cebuano was designated as the language for the TIDES surprise language dry run. This paper reports the results of the first 60 hours of our data collection and implementation effort

Cross-lingual C*ST*RD: English access to Hindi information

We present C*ST*RD, a cross-language information delivery system that supports cross-language information retrieval, information space visualization and navigation, machine translation, and text summarization of single documents and clusters of documents. C*ST*RD was assembled and trained within 1 month, in the context of DARPA’s Surprise Language Exercise, that selected as source a heretofore unstudied language, Hindi. Given the brief time, we could not create deep Hindi capabilities for all the modules, but instead experimented with combining shallow Hindi capabilities, or even English-only modules, into one integrated system. Various possible configurations, with different tradeoffs in processing speed and ease of use, enable the rapid deployment of C*ST*RD to new languages under various conditions